An image sensor, such as a camera, may be used to capture image data. Image data can be processed by an image processor to provide an image comprising a plurality of pixels. The initial image data representing the values of the pixels of an image may subsequently be encoded. An aim of encoding the image is often to reduce the amount of data that is used to represent the image. Therefore, the encoding of the image may involve some compression of the image. A lossless encoding algorithm may be used to encode the image without losing any of the information in the image. In contrast, a lossy encoding algorithm may be used to encode the image, and in doing so compress the image to a greater extent than it would be compressed using a lossless encoding algorithm. However, when a lossy encoding algorithm is used to encode an image, some of the information in the image may be lost.
There is often redundancy in the initial image data which represents the values of each of the pixels in the image, because there is often some correlation between pixel values in the image, e.g. for pixels which are located next to, or at least near to, each other in the image. For example, the colour of a group of pixels which all have the same colour can be compressed (losslessly) by storing the colour once along with some grouping information to represent the colour of all of the pixels of the group rather than by storing the colour multiple times, i.e. once for each of the pixels. The more redundancy there is in the image, the more the image can be compressed without losing any of the information in the image. A quantization parameter (QP) used by the encoder regulates how much detail is saved. When the QP used by the encoder is very small, almost all the detail in the image is retained. When the QP used by the encoder is increased, some of the detail in the image is aggregated so that the bit rate of the encoded image drops, but at the price of some increase in distortion and some loss of quality of the encoded image.
It may be beneficial for an encoder to provide an encoded image at a constant (or near constant) bitrate both across the image, and when the image is a frame within a video stream comprising multiple frames across the different frames of the video stream. A buffer may be used, and operated as a ‘leaky bucket’, which is emptied at a constant rate whilst being filled at a variable rate with encoded image data from the encoder. A rate controller may be used to adjust the QP used by the encoder in accordance with a target bitrate and such that the buffer is not depleted or saturated. The rate controller receives feedback from the encoder to indicate the number of bits that have been generated by encoding previous images and/or previous sections of an image currently being encoded. The rate controller may use the feedback to determine how the QP should be varied for encoding subsequent images and/or subsequent sections of an image currently being encoded.
The redundancy in an image may vary significantly from image to image as well as from one section of an image to another section of that same image. It may be that the image can be encoded only once (e.g. when the image is encoded in real-time), such that an image is not re-encoded if too few or too many bits are generated, relative to the target bitrate, by encoding the image. Furthermore, in order to reduce the storage requirements of the buffer and to keep the latency of the encoding of the image to an acceptably low level, the size of the buffer is constrained. Furthermore, larger variations in the QP used to encode different sections of the same image may result in a perceptually lower quality encoded image. Therefore, the rate controller might not allow the QP to vary by more than a threshold amount for the encoding of an image.
The constraints mentioned above can result in cases in which the rate controller does not optimally (or even nearly optimally) control the QP used by the encoder to encode the sections of an image (e.g. when the image is encoded in real-time). For example, the amount of detail in an image may be localised, e.g. the bottom quarter of the image may include more detail than the rest of the image. In this situation, the rate controller may be operating well within a bit budget for three quarters of the image, using up three quarters of the bit budget. However, when the encoder encodes the bottom quarter of the image containing a high level of detail, the rate controller is forced to increase the QP (compared to that which would be well suited for encoding the level of detail in the bottom quarter of the image) in order for the encoder to stay within the bit budget. Changing the QP in this way may result in a perceptually low quality encoded image.